This is nice work, got a lot further than my attempts to dig at this. Haven't had time to deep dive and/or think about the fiddling with SCons variables, but the evidence from the tests is promising. If this does go forward, someone will have to undo some of the doc warnings that were recently added to "don't do that" (nargs > 1, space separators). Also we should hunt down and tie in all the github issues that mention this problem, there are quite a few - at least the ones noted in #3436, there may be others too.

Looks like test: test/AddOption/args-and-targets.py is failing?
Can you take a look and resolve?

So this test line:

test.run('-Q -q -x A TARG1', status=1, stdout="A\n\\['TARG1'\\]\n")

seems to indicate A is showing up in the targets at the point the SConstruct checks for it after all (one of the old problems this is trying to address), because if it does, then B is added to targets and the result indicates that's being seen:

STDOUT =========================================================================
2c2
< \['TARG1'\]
---
> ['TARG1', 'B']

This ended up sitting for a long time. @bdbaddog - in terms of at least fixing #3798, would it be more palatable to pick just that part of this PR and submit it separately? The absence of an overridden process_short_options that understands we may not yet have seen the definition of the short option is clearly an omission - though we could also document that it just doesn't work, as came up in the issue. Right now,

AddOption(
    "-F",
    "--force",
    action="store_true",
    help="force installation (overwrite any existing files)",
)

Would fail with a "no such option", if you scons -F

This ended up sitting for a long time. @bdbaddog - in terms of at least fixing #3798, would it be more palatable to pick just that part of this PR and submit it separately?

yes.
So a part which is a partial fix and no downsides?

this is attacking two problems: single-char args, and the problem with args-with-options using space(s); figured it would be easier to get you happy with one at a time?